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ABSTRACT 



This paper examines comparative educational program 
evaluation. Suggested evaluative criteria and evaluation techniques 
and their weaknesses are discussed. An evaluation formula is 



proposed, and an example of its operation is provided, (DG) 
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PROCEDURES i- OR CO”. PARING I M STRUCT I ORAL PROGRAMS 1 



S t e p ! i e n K 1 e i n 

UCLA Graduate School of Education 



The utility of various techniques for comparing the effec- 
tiveness of different instructional programs is becoming an 
important topic in the field of educational evaluation. As 
P o p h a m ( 1 9 C 9 ) points out , one reason for this increased interest 
in program comparison techniques is that instructional programs, 
promising specified changes in learners, are becoming essentially 
replicable. This means that wo now have the opportunity to select 



from among competing instructional programs. Pressure for develop- 
ing valid and informative program comparison techniques also is 



emetnating from major educational funding sources. These funding 
sources want to know which programs (or program components) are 



effective and which are not so that they can allocate their support 
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accordingly. 

The very range of programs currently on the market, however, 
has already made it difficult for decision makers to select among 
them. This selection process is further complicated by the fact 
that alternative programs, even those designed for the same general 
content area, frequently focus On different objectives and vary in 
their success in achieving these objectives. Competing programs, 
also, often require different amounts and/or kinds of resources, 
personnel, and time. 
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Ffi c t nrs to Consider 

This situation has recently led a number of researchers to 
suggest various methods for making program comparisons. It is 
necessary, however, to examine the major program factors any 
effective program comparison technique should take into considera- 
tion, before, one can make an informed decision about their rela- 
tive merits. Six of these major factors are as follows: 

(1) Success on the Objectives 

(2) Relative Importance of the Objectives 

(3) Number of Objectives and their Overlap 

(4) Time Spent in Achieving the Objectives 

(5) Number and Kinds of Students Involved 

(6) Costs and Resources 

Success, on the Objective* . It is generally agreed that the 
foundation of ary good educational evaluation system is a clear 
statement of relevant program objectives. Further, these objec- 
tives should be stated in a way that makes them measurable. Once 
this is done, it is possible to assess student performance with 
respect to the degree to which the objectives have been obtained. 

It is obvious, therefore, that when the other variables noted above 
are held constant, one program is considered better than another 
when it has more success on its objectives. 

Im£^rtance of the A second way we can 

judge whether one program is better than another is in terms of 
the importance of the objectives it achieves. Thus, a program 
that has a moderate degree of success on very important objectives 
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may be more valuable than one that has a high degree of success 
on objectives which are relatively less important. For example, 
a program that results in 50% of the students being able to read 
at grade level would probably be more valuable than one that results 
in 90% of them being able to name all the states in the Union and 
their capitals. 

Any system for comparing the effectiveness of programs or 
program components, therefore, must consider the relative importance 
of the objectives being attempted. But it also must be remembered 
that although the d i ffj c u 1 t y of achieving a given objective is 
likely to be positively related to the objective's importance , the 
two are net synonymous. Further, difficulty in and of itself has 
no social significance. For example , of the two objectives listed 
below, the first is probably much more difficult than the second 
to achieve. The second one, however, is likely to be considered 
more important to achieve: 

(1) 90% of all high school graduates can find the 
square root of any 5 digit numeral. 

(2) 90% of all high school graduates can balance a 
cheeking account. 

Nurnbev_ of anii Qperiap . Another area Of 

major concern in the comparison of programs is the number of objec- 
tives and their overlap. Where all other things are constant, we 
can consider a program that achieves more objectives than another 
program to be more desirable. As was mentioned earlier, however, 
programs often differ in the number of objectives they attempt to 
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echicve, and a problem will arise if we attempt to compare a pro- 
gram having moderate success on many objectives with a program 
having a great deal of success on just a few objectives. 

This problem is by no means sol ved 5 however, if program 
comparison is limited exclusively to objectives that are common 
to both programs. For example, one program may have substantial 
success on the overlapping objectives but it focuses e x c 1 u s i v e 1 y 
on these while a second program may have moderate success on many 
objectives including those that overlap with the first program. 

In other words, a program nut y look better than it actually is 
simply because it attempts to accomplish less than another. Any 
system for comparing programs, therefore, must adjust for the 
number of objectives attempted in each program. 

Tijne Spent in Object . All other things 

constant, the less time a student spends in achieving an objec- 
tive, the more effective the program. There are, of course, 
several reasons for this, such as the fact that time means money, 
after awhile pupils become bored with any subject, and it becomes 
harder to teach, and time used for one purpose cannot be used in 
achieving other objectives. 

Numjper of §J^udentB_ . A fifth way we can assess whether one 
program is more effective than another is in terms of whether it 
improves the performance of more pupils. Unlike the examples in 
simple statistical texts, however, the practitioner often encoun- 
ters situations in which he must compare one program that achieves 
success with a few students versus one that achieves somewhat less 
success but has many students. It is apparent, therefore, that 
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any good program comparison method should control for the number 
of students involved since this will directly influence the pro- 
gram's impact. This is especially true when comparing new pro- 
grams since a program frequently is more successful when tried 
out in a few classes than when it is used throughout a district. 

Program Cop ts. All other things constant, one program is 
more effective than another if it costs less to operate and still 
achieves the same degree of success. In comparing programs, how- 
ever, one must take into consideration the fact that although one 
program may result in higher levels of student performance than 
another and in less time, it is only able to achieve this at a 
much higher per pupil cost. 

The cost issue also reflects on the fact that school and 
government administrators want more information than just student 
performance in order to make decisions about programs. In other 
words, they want to know about the relative importance of objec- 
tives, the number of objectives, the number of pupils involved, 
and the time and costs needed to achieve the objectives. All too 
often they must rely on subjective and faulty appraisals of these 
latter factors since the program evaluators have only reported on 
average student performance . Thus, in order to provide a more 
valid basis for making program decisions, evaluators should provide 
administrators with a consistent and systematic approach for 
considering all the major program factors that influence such 
decisions. 

Earlier Techniques for Comparing Programs 



If we use the six factors noted previously as a set of 
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standards for any program compari son technique, then we must 
conclude that a groat many of the current techniques are too 
simplistic to provide any worthwhile comparative information. 

In determining which of two or more competing instructional pro- 
grams is best, for example, the decision is often made on such 
extraneous considerations as packaging of materials, their format, 
and their illustrations. While we may keep such considerations 
in mind, it should be obvious that they provide no real informa- 
tion in terms of program effectiveness in meeting objectives. 
Similarly, decisions concerning which program to select are often 
based on purely subjective considerations such as style or philo- 
sophy in the program's treatment of content. Again, such consi- 
derations are almost worthless in terms of providing information 
which can be used to decide upon program success in meeting objec- 
t i v e s . 

Independent Tests_ , One technique for comparing programs that 
is employed quite often involves the use of a nationally normed 
standardized test across all the different programs being compared. 
There are several advantages in applying this method. First, this 
type of measure is constructed without intentional bias for one pro 
gram over another and, hopefully, will not favor the content of one 
program over another. Further, the same measure is given in all 
programs under the same conditions and there is a set of norms 
and other related information available with the instrument to 
help in interpreting results. Third, the test construction proce- 
dures used in these instruments are generally good compared to 
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locally constructed instruments. Finally, they are relatively 
inexpensive and readily understood by laymen. 

There are, however, many disadvantages in using the indepen- 
dent test technique. For example, such tests ideally will not be 
biased in favor of one of the programs being compared; they may 
inadvertently favor one program over another in terms of emphasis 
on the objectives measured (Klein, 1970). Further, such tests 
may not cover all t h e relevant objectives in each program or even 
all those that overlap programs. More importantly, the methods of 
test construction and score reporting of most nationally normed 
instruments do not emphasize providing information about student 
p e r f o riri ance on specific objectives (Klein, 1971), th ey do not 
consider differences in the number of objectives covered, and they 
neglect some of the other important considerations that were 
discussed in the introduction to tin’s paper. 

Program Unique ffests . A different method of program compari- 
son is discussed byWolf (1968). In this technique, each program 
has its own unique test for its specific objectives. Program 
comparisons are made solely on the basis of the degree to which 
students in each program achieve the objectives for their program. 
The advantages of this approach over the use of the standardized 
tests technique are that tests are designed specifically for each 
program and students do not have to take test items dealing with 
material that they have not covered. 

Despite these advantages, this method still has several short 
comings. For example, it offers no techniques for handling differ 
Wees in test difficulty, number of objectives covered, importance 
of the objectives, or the time spent on them. 
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To meet the short c. cm i ngs in the methods discussed above, other 
techniques have been developed which, in a very restricted sense, 
provide for better comparison of programs. Two of these methods 
which we will discuss, while providing some advantages over 
earlier methods, are still extremely limited in that they require 
that the programs being compared either have common objectives or 
objectives which overlap to a high degree. 

Frogr am F roe T ec t -i n <7 , One of the more recent methods for 

making comparisons is called Program Free Testing (Wolf, 1968). 

This method involves using a test which has been constructed to 
assess those objectives or program elements common to all the 
programs being compared. The first step in this method is to 
compile a list of those objectives which are common across pro- 
grams and then to construct test items for these objectives. 

After the test has been administered, the average scores of students 
in the various programs are compared. 

The advantages of this method are that a common measure is 
used across programs and it focuses on specific but common pro- 
gram objectives. By focusing on common objectives, however, this 
method fails to consider those objectives which are unique to each 
program; and these latter objectives may be more important than 
the common objectives. Further, the method is highly dependent 
upon a large degree of overlap between program objectives, and 
thus, if this overlap is slight, then only a small aspect of each 
prog, am is considered. Finally, like the other methods discussed. 
Program Free Testing still fails to consider such factors as 
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differences in the relative importance of objectives across pro- 
9 rams and the number of objectives each program is trying to 
achieve. 

Program Fa i y Testing . A second recent method of making pro- 
gram comparisons involving overlapping objectives has been 
developed by Popham (1969). This method. Program Fair Testing, 
has one advantage over the others in that it at least recognizes 
that programs being compared may not always have totally common 
objectives. Popham suggests that the objectives of each program 
be identified and then grouped according to whether they are 
common to the programs being compared, unique to one program, or 
unique to another program. Tests are then constructed for each 
set of objectives so that three or more subtests are developed; 
that is, one subtest for the objectives common across programs, 
and one for the unique objectives of each program being compared. 
Students in each program receive two tests, i . e . , the test for the 
common objectives and the test for the objectives unique to their 
program. Based upon performance of the various groups of learners, 
and considering the relative values of each set of objectives, a 
comparison can be made to determine which of the programs should 
be sel acted . 

This technique offers several advantages in that it uses a 
common measure across programs, it pays attention to unique objec- 
tives within each program, and recognizes differences in the rela- 
tive importance of objectives. The method is again limited, how- 
ever, in that it still relies heavily on overlapping objectives. 
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In addition, the procedure makes no provision for s_y_s tematical 1 y_ 
handling differences in the relative importance of objectives, 
the number of objectives covered, and related factors, although 
Popham does state that they are important considerations. 

Unanswered Quo s t i on s 

Of the techniques for program comparison that we have 
discussed so far, the Program Fair and Program Free Testing provide 
better information than previous methods. Roth are limited in 
scope and application, however, as evidenced by the fact that they 
are appropriate only when program objectives are common or at 
least overlap to a fairly high degree. In addition, they both 
leave unanswered many questions which must be addressed if we are 
going to perform realistic comparisons of different programs. 

For example, what are we to do if the objectives of the programs 
being compared do not overlap to a very high degree? Or if they 
share no common objectives? What if we are faced with a more 
complex situation such as when one program has a total of 25 pupils, 
takes all year, and has ten objectives; while a second program 
has 500 pupils for a semester and only has five objectives, and 
only two of these overlap the objectives of the first program? 

As we have seen, comparisons based on independent tests or 
just success in meeting program objectives do not answer the 
questions raised above, nor do these methods adequately consider 
the many other factors we have identified as being essential to 
valid program comparisons. As an attempt to move one step forward 
in resolving this problem, the remainder of this paper is devoted 
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to a description of how the six basic factors we have identified 
might be incorporated 'into a potentially more effective program 
c o rn p a r i s o n technique. 



General Formula for Determining Program Effectiveness 

Any method of comparing programs should take into considera- 
tion all the major factors that might influence the relative 
effe c tiveness of one program over another. The formula below 
depicts the general relationships of these program character!* s- 
t i cs : 



(S) (N) 

err 



E 



Where: S - the sum of the total weighted scores for all 
the program's object! ves (where the scores 
on each objective are some function of 
success on the objective relative to its 
judged importance). 



N = number of pupils in the program, 

T = amount of pupil time spent in the program, 
C = cost of running the program. 

E = general program effectiveness . 



The rationale for this formula is that general program 
effectiveness will increase if one or more of the following varia- 
bles increases: number of objectives, success on the objectives, 
relative importance of the objectives, number of students in the 
program; or if the pupil time and/or program costs decrease. The 
key to this formula is that the relative importance of objectives 
across programs is based on a common scale and thereby eliminating 
the necessity of overlapping objectives. How this formula can be 
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used for comparing programs will be discussed after the considera- 
tion of the general problems inherent in calculating the appro- 
priate numbers of S, N. T, and C. 

P rob 1 ems a nd G u i delines Associated 

with the Ge n eral F o rm u 1 a 

Like most general formulas, the one suggested above has its 
problems. Some of these problems are applicable to the field of 
evaluation in general while others are especially central to the 
issues associated with comparing programs. Some of the important 
problems associated with each of the six major program character- 
istics are discussed below along with some suggestions for how to 
deal with these problems in using the general formula. 

Suqo&ss on the QbJ_eo^ives_ , Some of the major problems 
associated with comparing programs in terms of their relative 
success on the objectives are as follows: 

1, Type of score reported. The number of test items and 
the method of measurement often varies across objec- 
tives both within and between programs. It is neces- 
sary, therefore, to convert performance on different 
objectives to a common scale, such as the average 
percent correct, average number of students per class 
achieving mastery, or the average time it takes a 
student to achieve mastery. There are, of course, 
other scales that might be used and the choice of 
which one to use might have a profound influence on 
the conclusions drawn from the data (see Appendix A 
for an ex am pi e ) . 

2, Difficulty. Different assessment instruments, even 
for the same objective, often vary in their diffi- 
culty, Thus, one program may appear to be more 
successful than another program in reaching its 
objectives simply because it is using easier measures. 
Thus, whenever possible, the same set of Items should 
be used to assess performance on a given objective 

if that objective is applicable to different programs. 
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3. Scaling, The difficulty of "items" within a test 
for a given objective usually are not equal nor do 
they have equal intervals of difficulty between them. 

For exainpl e , it may be easier to improve 10 points 
near the low end of the score distribution than near 
the high end. 

4. Validity. The more valid a measurement instrument 
is ( i . e . , actually measures the objective rather 
than an approximation of it), the more sensitive 
it is to changes in student performance. Thus, 
differences in measurement validity across objec- 
tives may create the false impression that some 
objectives are achieved more than others when the 
real reason is that the instruments used varied in 
their ability to assess the precise level of student 
performance, 

5. Reliability, Misleading differences and similari- 
ties in student performance across objectives and 
programs can also be caused by variations in the 
consistency with which instruments measure differ- 
ent objectives. The reliabilities of instruments 
used in most evaluation studies, however, are 
usually sufficient to minimize this problem, 
especially when decisions about program effective- 
ness are based upon group rather than individual 
scores . 

Relative XmiDortanoe of Qbjjatives^ , There are several ways 
of determining the relative importance of objectives. One of the 
simplest and most efficient of these techniques is to have a 
panel of relevant judges rate the total set of objectives (i.e., 
for all the programs being compared) on a common scale (e.g., from 
"5" = very important down to "0" = very unimportant, or irrelevant) 
or by having them distribute a total set of points among all the 
objectives to be evaluated. 



Number oJ_ Ohj_e_otivea^ . The present general formula may be 
biased in favor of a program that has moderate success on many 
easy but relatively unimportant objectives versus one that has 
only limited success on a few very important objectives. In 
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typical situations involving program comparisons, however, the 
most likely source of major differences in the number of objec- 
tives is the level of generality at wh i c n they are written. I h i s 
problem can be partially rectified by redefining and explicating 
objectives so that they are all written on the same level. The 



method noted above of distributing a total set of points among 
all the objectives to be evaluated (rather than assigning 
absolute weights) also would help to counterbalance the effects 
of differences in the generality in which objectives are stated 
across projects. The reason for this is that the sum of the 
weights assigned to a set of relatively specific objectives 
would presumably be the same as that assigned to the more general 



objective from which the specific ones are derived. 



Time Sj^enii in Objectives . Collecting valid infor- 

mation about the pupil time spent on achieving program objectives 
is at best a difficult task. One reason for this is that teachers 
do not have the time to keep track of this information on a day 
to day basis for a class as a whole let alone for individual 
pupils or objectives. It is possible, however, for teachers to 
make a fairly accurate estimate of the general amount of time the 
class as a whole has spent in a particular program. Such time 
estimates should probably be limited to time spent in school and/ 
or under direct supervision d y the teacher rather than try to 
judge and include such things as homework time. These estimates 
should be in terms of the average number of hours (or days or 
weeks) the typical pupil spends in the program. 
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h lumber of Stud ents , There are several ways of indicating just 
how many students are in a given program, such as the numbe^ of 
students listed as enrolled, average daily attendance, and the 
number of students tested or measured for evaluation purposes. 

The latter method, however, has several advantages, e.g., it 
reduces complicated record keeping and it focuses on pupil progress 
at the time the evaluation is being conducted. This approach also 
allows the general formula to correct for certain possible biases 
in the data, such as excluding potentially poor achievers from 
taking the tests so as to make the group's average higher than it 
really is (the general formula takes this into consideration in 
the sense that program effectiveness is a function of both success 
on the objectives and the number of students tested). Despite 
these corrections, it would still be a good idea to include a 
listing of the students tested relative to those enrolled (perhaps 
broken down by age, sex, or some other salient characteristic) to 
ensure that the students tested are truly representative of the 
students who are supposed to be in the program, 

Pyogram Co8t8_ . The various methods used for establishing 
program costs is not the focus of this paper. This and related 
issues have been discussed by Hartley (1968) and others, however, 
some factors which should be considered in establishing program 
costs are; 

1. Long-run versus short-run costs; e.g., can the same 
equipment be used the following year and therefore 
permit its cost to be spread? 
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2. Cost efficiencies in having more or less students in 
the program; i.e., is the per pupil cost influenced 
by expanding or contracting the program? 

For simple program comparisons, it may be more efficient to limit 
the cost analysis to easily computed variables such as materials, 
staff, etc. 

Spaoial P p ope .' I'ics and L i in i t a t i- o n e . The general formula for 
computing program effectiveness has certain properties that in 
some ways enhance and in other ways limit its utility. One of 
these properties deals with the fact that some of the factors 
discussed above are not independent of one another. The number 
of students enrolled in a program and that program's cost, for 
example, are generally positively related to each other. Similar- 
ly, the number of program objectives relative to their weighted 
importance and the amount of time spent on trying to achieve them 
are also likely to be related to one another. Although the general 
formula handles this problem (e.g., doubling program costs while 
doubling the number of students would have no impact on the value 
for E), it does not make any provision for the relative importance' 
of various factors. In other words, some decisionmakers might 
like the formula to give more emphasis to success on the objectives 
in determining program effectiveness while others might like to 
see a greater weight given to program costs. Multiplying the 
values for S, N, T, and C in the formula by the desired weights 
would, of course, permit such differential emphasis on major pro- 
gram characteristics. 
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Tho 9 a no ral formula also contains a very mixed bag of measur- 
ing scales j e . g , , hours, weighted test scores, and cl o 1 1 a r s . Thus, 
s m a 1 1 variations in one factor ( such as costs) might have to be 
offset by rather major changes in another factor (such as weighted 
success) in order to obtain the same value for "program effective- 
ness" (E). It should be noted, however, that the purpose of the 
formula is to compare programs in terms of their relative effective - 
ness and not to make judgements about their general effectiveness 
in any absolute sense. When it is used properly, therefore, these 
differences in scaling are matched between programs (i.e,, costs 
of one program are considered relative to costs in another, hours 
spent in one program are compared to hours spent in another, etc.). 
The next section of this report, computational procedures, discusses 
in more detail this emphasis on the formula's use for determining a 
program's relative effectiveness. 

Computational Procedures and Numerical Example 

Listed below are the basic steps used in comparing the effec- 
tiveness of alternative programs. A numerical example accompanies 
these steps to illustrate the necessary computations in employing 
the general formula for the comparison of two programs. 

1. Operationally define each of the four major components of 
the formula, i.e., total weighted score on the objectives, number 
of students in the program, total time spent in the program, and 
total program costs. These definitions must, of course, be the 
same for all the programs being compared. The operational defini- 
tions for the numerical example are as follows: 
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a. Total weig_ht_e_d sco re( s) . The series of calculations 
nc“tfd e d~TcT^o b t S' T"n” this’ score are as follows: 

(1) Compute each student's score on each objective 
in terms of the degree to which he has mastered 
the objective (e.g., 10%, 20%, etc,)- 

(2) Compute the average of these proportions on each 
objective for all the students in each program. 

(3) Multiply the average proportion of success on 
each objective by its weighted importance. 

These weights are determined using the proce- 
dures described on page 2_3 of this report, 

(4) Compute the sum of these weighted averages for 
each program. 

The raw test data for this example appears in 
Appendix B. The summary results for each program 
appear in the table on the next page. 

b. Number of students. For the purposes of the present 
example, this will be the number of pupils tested in 
each program, i,e,, n ^ - 20, n g - 25, and N = 45. 

c. Time. The typical student spent 1_0 weeks in program 
#1 and 15 weeks in program #2. 

d. Cost. The total cost of running program #1 was 
$750.00 and the total cost of running program #2 
w a s $ 1 ,000.00. 
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Table 1. COMPUTATION OF TOTAL WEIGHTED SCORES 
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2, Compute e for each program as follows: 



(Sj) ( n ^ ) 

(tj) Tc 3 ) 



e r 



{ s g ) ( n 2 ) 



(t 2 ) (c 2 ) 



In terms of the numerical example, the values for and e 2 
are as follows: 



(1203) (20) 
(15) (2000) 



e, s .80 



(1153) (25) 



e 2 s 2,88 



( 10 ) ( 1000 ) 



3, Compute E as follows: 

e . + e 2 “• E thus, _JK) + 2,88 » 3.68 = E 



4. Compute the following ratios and convert to % 

e e 

e l 



100 X — = relative effectiveness 
E of program £1 



,80 

100 X K 22% 

3 . 68 



100 X 
2,80 



100 X 



3.68 



= relative effective- 
ness of program # 2 



78% 



It is apparent from the foregoing results that program II is more 
effective than program j_. It should be remembered, however, that 
tin’s conclusion might be quite different if one wished to differen- 
tially weight the values of S, N, T, and C in the general formula. 
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$ u mm ary & Cone 1 u s ions 

The first portion of this paper discussed six important factors 
that should be considered in comparing the relative effectiveness 
of educational programs, e . g . , the students' performance on the 
programs' objectives and the costs involved in achieving these 
objectives. The second portion of this paper examined the relative 
utility of previous techniques .that have been used for making such 
comparisons. The major weaknesses of these techniques is that 
they fail to take into account several of the important factors 
that should be considered as well as place too much on the 
comparability (or at least partial overlap) of objectives across 
programs . 

The final section of this paper presented one way in which 
the foregoing problems might be handled. This approach uses as 
its common denominator the relative importance of objectives 
across programs, It also considers sy s temati cal 1 y each of the six 
important factors that should be taken into account when one is 
comparing the relative effectiveness of various programs. Thus, 
the approach suggested in this paper is qualitatively different 
than previous ones since its focus is upon the kinds of factors 
that influence decisions regarding program selection. Whether or 
not the particular procedures outlined in this paper are adopted is, 
therefore, of relatively secondary importance. What is critical is 
that the suggested procedures highlight the kinds of factors that 
must be taken into consideration if one wishes to make valid 
program comparisons. 
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Appendix A : 


Data for Numerical Example 




PROGRAM I 



Pupil 


A 


B 


Obj ective 

C 


'd 


TOTAI 


1 


50 


40 


20 


60 


170 


2 

3 


. 60 


70 


60 


80 


270 


' '100 


100 


90 


90 


380 


4 


80 


90 


90 


100 


360 


5 


60 


50 


50 


70 


230 


6 


SO 


70 


60 


80 


290 


7 


70 


90 


100 


60 


320 


8 


70 


80 


50 


90 


290 


9 


60 


60 


60 


60 


240 


10 


50 


50 


40 


60 


200 


11 


40 


40 


30 


30 


140 


12 


100 


100 


90 


100 


390 


13 


90 


100 


100 


100 


390 


14 


SO 


100 


90 


90 


360 


15 


90 


100 


100 


90 


380 


16 


40 


30 


50 


60 


180 


17 


50 


- 50 


30 


50 


160 


18 


70 


50 


60 


' 60 


240 


19 


80 


70 


40 


90 


280 


20 


- 100 


100 


100 


100 


400 


Total % 


1420 


1420 


1310 


1520 




Correct 

Weighted- 


5,0 


4.5 


4.0 


3.5 





Importance 



eric 
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Appendix A : Data for N u in e r i c a 1 Ex a m p 1 e 

PROGRAM IT 



Pupil 


A 


B 


Objective 


D 


E 


1 


20 


80 


60 


70 


40 


2 


30 


30 


10 


20 


20 


3 


50 


60 


20 


60 


10 


4 


. *20 


40 


40 


50 


60 


5 


80 


80 


60 


90 


70 


6 . 


70 


70 ■ 


70 


80 


SO 


7 


70 


70 


100 


40 


60 


8 


60 


40 


60 


30 


40 


9 


30 


20 


20 


20 


20 


10 


100 


80 


90 


90 


100 


11 


80 


70 


70 


100 


70 


12 


90 


100 


90 


90 


80 


13 


100 


90 


100 


100 


90 


14 


80 


90 


70 


90 


60 


15 


50 


50 


70 


40 


50 


16 


100 


60 


60 


50 


50 


17 


20 


10 


40 


20 


50 


18 


70 


80 


60 


90 


40 


19 


30 


‘ 50 


40 


30 


20 


20 


30 


20 


20 


40 


10 


21 


80 


60 


60 


40 


40 


22 


40 . 


30 


50 


20 


70 


23 


100 


100 


100 


90 


90 


24 


90 


100 


100 


90 


80 


25 


50 


40 


30 


40 


20 


Total % - 


1530 


1530 


1490 


14S0 


1320 


Correct * 












Weighted 

Importance 


5.0 


4.0 


4.0 


3.5 


3,0 



/ 






TOT 

27 

11 

20 

21 

38 
37 
34 
23 
11 
46 

39 
45 

• 48* 
39 . 
26i 
331 
14C 
34C 
17C 
12C 
28C 
21C 
48C 
46C 
18C 



